Characteristic region extraction device, characteristic region extraction method, and characteristic region extraction program

ABSTRACT

The conventional image retrieving method is not satisfactory in accuracy of retrieval. There has been a demand for a general-purpose method capable of rapidly retrieving characteristic parts. The object is achieved by acquiring image data representing an image with pixels arranged in a dot matrix pattern, detecting the edge pixels of the image according to the image data, extracting the edge pixel as the characteristic point when a pattern formed by the detected edge pixels and their neighboring pixels is similar to a prescribed object to be extracted, and defining as the characteristic region the prescribed region in the image which has the extracted characteristic points in large number.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a device, method, and program forextracting a characteristic region.

2. Description of the Related Art

It is often necessary to retrieve images (entirely or partly)photographed by a digital camera or to retrieve adjoining positions atwhich images are stitched together to form a panorama photograph. Tomeet this need, there have recently been proposed several methods forretrieving characteristic parts in an image. (See, for example,“Procedure for extracting characteristic points by DRA (DecrementalResampling Algorithm)” [on-line] searched Nov. 20, 2002 at the followingInternet URL.http://www.koshi-lab.sccs.chukyo-u.ac.jp/research/picasso/side/dra.html

OBJECT AND SUMMARY OF THE INVENTION

The conventional image retrieving method mentioned above is notsatisfactory in accuracy of retrieval. There has been a demand for ageneral-purpose method capable of rapidly retrieving characteristicparts. The method disclosed in the non-patent document mentioned aboveis intended to extract features by extracting an edge of an image,deleting edge pixels meeting specific conditions which depend on certainedge pixels and their neighboring edge pixels, and repeating thedeletion until the number of re-sampling points is reached. This methodrequires a large amount of computation and is slow in processing speed.Moreover, this method is not generally applicable because it is based onthe assumption that a set of edges forms a closed curve. In addition,this method is poor in accuracy of retrieval.

The present invention was completed to address the above-mentionedproblems involved with the conventional technology. It is an object ofthe present invention to provide a device, method, and program forextracting a characteristic region, which are intended to extract acharacteristic region rapidly and accurately in general-purpose use.

The present invention achieves the above-mentioned object by extractingan edge in an image, judging whether or not its shape coincides with theshape of the object to be extracted, and defining the coinciding part asthe characteristic region. Therefore, the device according to thepresent invention has an image data acquiring unit and an edge pixeldetecting unit, so that the former acquires image data and the latterdetects edge pixels in the image. Moreover, the device according to thepresent invention has a characteristic point extracting unit and acharacteristic region defining unit, so that the former judges whetheror not the detected edge pixels and their neighboring pixels are similarto a prescribed pattern to be extracted.

In this way, the edge pixels of the part whose edge shape is similar tothe prescribed pattern are extracted as the characteristic point. Theedge pixels extracted as the characteristic point become the candidatefor the characteristic region. The characteristic region defining unitdefines the prescribed region having many characteristic points as thecharacteristic region. This characteristic region is a verycharacteristic region which contains many characteristic points thatform the prescribed pattern to be extracted. In general, pixelsconstituting photographic images are complicated and they form variousedges. However, it is possible to easily specify a very characteristicregion in an image and extract it by noting that edges contain manycharacteristic points.

To be concrete, since a characteristic point similar to the pattern tobe extracted is extracted, there are usually more than one candidate forthe characteristic point. However, the regions containing thesecharacteristic points are very limited. Consequently, if the regionhaving many characteristic points in the image is defined as thecharacteristic region, then it is possible to restrict the candidate tobecome the characteristic region from the image even though the imagemicroscopically contains many objects to be extracted. Thus, it ispossible to easily extract the characteristic part from the limitedcandidates. This constitution permits highly accurate image retrievaland image matching with very few errors.

In other words, the characteristic region to be appropriately extractedcontains the prescribed pattern to be extracted and hence thecharacteristic points in this region are certainly extracted simply byjudging whether or not the neighbor of the edge pixels coincides withthe pattern. However, simple judgment on coincidence with the patternmay lead to extraction of the part which forms by chance the pixelssimilar to the pattern but is not the characteristic region. Thisresults in extraction with many errors, particularly in the case ofcomplex pixels forming an image of natural photograph.

According to the present invention, the foregoing problem is addressedas follows. When the pattern formed by edge pixels and their neighboringpixels are similar to the pattern to be extracted, the edge pixels areregarded as the characteristic point and the extracted region havingmany characteristic points is regarded as the characteristic region.This characteristic region is the part which contains many objects to beextracted. It is a very unique region in a natural photograph or thelike, and hence there are very few possibilities of similar regionsexisting in the same image. Thus, this characteristic region can beextracted with a very high degree of accuracy.

Particularly, when a specific region is searched for in a portion of animage (as in the case of retrieving those parts coinciding with eachother in the ends of two images to be stitched together into a panoramaphotograph), there is almost no possibility that such characteristicregions exist in large number in each of the images to be stitchedtogether. However, if two images are stitched together, a characteristicregion should exist almost certainly in one part of each image.Therefore, extraction of a characteristic region according to thepresent invention makes it possible to retrieve the stitching parts fromthe two images with a very high probability. In other words, it ispossible to extract the characteristic region very accurately. Imageretrieval with higher accuracy will be possible by imposing constraints,such as selecting the characteristic region from one part each in theright and left ends of images to be stitched together in which thenumber of characteristic points is maximum.

Also, according to the present invention, detection of edge pixelspermits extraction of characteristic points as candidates in thecharacteristic region. In this case, the number of characteristic pointsin the entire image is very small even in a natural image in which eachedge has a comparatively large number of pixels. Moreover, after thecharacteristic points have been extracted, their number can be used asan index to determine the characteristic region. In this way it ispossible to extract the characteristic region by inspecting a much lessnumber of objects than the pattern matching in which a prescribedpattern is extracted by sequentially comparing pixels in an image. Thisleads to a high-speed processing. Moreover, as compared with theabove-mentioned conventional process which requires various processes tocompute as many edge pixels as the number of re-sampling points, themethod of the present invention can extract the characteristic regionwith a much less number of steps. This permits a high speed processing.

Detection of edge pixels is accomplished in various ways in general use.Extraction of the characteristic point only needs comparison of the dataobtained by edge detection with the prescribed object to be extracted.This method is applicable to any kind of image data irrespective ofimages and edge pixels. The fact that the characteristic region isdetermined by the number of characteristic points permits the method tobe applied to any image data irrespective of images and edge pixels.Therefore, the method of the present invention does not assume that edgepixels connected together should form a closed curve, unlike theconventional one mentioned above; it is a general-purpose method thatcan be applied to images of any kind.

Incidentally, the unit to detect edge pixels is not specificallyrestricted so long as it is capable of detecting edge pixels in theimage data. It includes such filters as Sobel filter, Previtt filter,Roberts filter, and Laplacian filter. They are designed to compute thegradient of gray level of certain pixels and determines that such pixelsconstitute an edge by judging whether or not the computed gradientexceeds a prescribed threshold. The threshold may be adequately adjustedaccording to the gradient of pixels to be detected as an edge. Norestrictions are imposed on the color of the image data to be detected.For an image in which each color is represented by its gray level, edgesmay be detected for each color. Alternatively, it is possible to computethe luminance of each pixel and then detect the edge having the computedluminance.

The unit to extract characteristic points should be able to judgewhether or not the pattern formed by edge pixels and their neighboringpixels is similar to the prescribed object to be extracted. For simpleand rapid processing, this unit should preferably be so constituted asto make judgment by assigning “1” to the edge pixels and “0” to thenon-edge pixels, because pixels adjacent to edge pixels are either edgepixels or non-edge pixels.

In addition, for simple and rapid processing, it is desirable to getready a predetermined pattern as the prescribed pattern to be extracted,and it is also desirable to store a filter of dot-matrix type. (Thisfilter forms a pattern for extraction with a filter value to indicateedge pixels and a filter value to indicate non-edge pixels.) The storedfilter permits easy comparison between the edge detecting data (in which“1” is assigned to edge pixels and “0” is assigned to non-edge pixels)and the prescribed pattern to be extracted. Comparison may beaccomplished in various ways, e.g., by superposing the filter on theedge detecting data and performing AND operation on the filter value andthe value (1 or 0) indicating the edge pixels or non-edge pixels.

The comparison should be able to judge whether or not the pattern formedby edge detecting data is similar to the prescribed pattern to beextracted. Comparison may be made by judging whether or not the numberof coincidences between the filter value and the edge detecting data islarger than a prescribed number. A typical method may be as follows. Ifa coincidence between the edge pixels of the edge detecting data and thefilter value to indicate the edge pixels exists at two or more placesaround the edge pixel at the filter center, then the edge pixel at thefilter center is regarded as the characteristic point. The condition of“more than two places” is a mere example; it may be changed to adjustthe processing speed according to the desired accuracy of extraction andthe number of characteristic points.

It is possible to variously change the prescribed pattern to beextracted which is set up when the characteristic points are extracted.It is only necessary to distinguish one region and another if a certainregion in one image contains many patterns to be extracted, and tocharacterize the region for discrimination and extract the characterizedregion as the characteristic region. A pattern meeting this requirementis one which has edges forming an angle larger than 90° and smaller than180°. In other words, it is difficult to characterize a featureless part(such as sky or single-color wall) in an image.

In order to characterize an image, it is desirable to extract a partwith characteristic edges (in a natural image) rather than a part withfeatureless edges. A part with edges forming many angles should beextracted as the characteristic part in an image. The angle between twoedges should be larger than 90° and smaller than 180°, because an actualimage rarely has two edges forming an angle of 90° and two edges formingan angle of 180° are straight and meaningless. Moreover, two edgesforming an angle of 0° to 90° should be excluded because they areaccompanied by many noises. The present inventors found that many anglesare detected from many images if the pattern for extraction has an angleof 135°.

Filters of varied size will be available that form the pattern to beextracted. They include that of 3×3 pixels or 5×5 pixels, for example. Aplurality of filters differing in pattern may be prepared previously. Adesirable filter for high-speed accurate processing is a 3×3 pixelfilter in which adjoining four pixels are assigned to the filter valuefor edge pixels and another adjoining four pixels are assigned to thefilter value for non-edge pixels.

The 3×3 pixel filter is small but enough to represent a pattern with acertain area, and hence it permits comparison with a few steps. If it isassumed that the central pixel of the filter represents an edge and theadjacent four pixels represent edge pixels, then the three edge pixels(with none of them at the center) and the two edge pixels (with one ofthem at the center) form an angle of 135°. This pattern with only 3×3pixels forms any angle from 90° to 180° and permits extraction of manycharacteristic points. This contributes to high-speed accurateprocessing.

The unit to define a characteristic region is only required to be ableto define as the characteristic region the prescribed region in theimage which has the extracted characteristic points in large number.Such a characteristic region defining unit may be realized by previouslyestablishing the size (the number of horizontal and vertical pixels) ofthe characteristic region, assigning the region of this size to theregion having the characteristic points in large number, and defining asthe characteristic region the region which has the maximum number ofcharacteristic points.

The region to be defined as the characteristic region may be establishedby dividing the image into two or more regions each composed of aprescribed number of pixels and selecting a specific region in which thenumber of characteristic points exceeds a prescribed threshold value. Inthis case, it is further possible to select a limited region from theregion having a large number of characteristic points. This isaccomplished by, for example, computing the average value of the edgegradient of pixels contained in each region and defining the region witha high average value as the characteristic region. This process involvesthe summing up of the edge gradient for individual pixels in the region;however, this process can be accomplished faster than the process to beperformed on all images, because the region consists of regions inlimited number selected from all images.

The present invention is also applied to the retrieval of positions forstitching two or more images to produce a panorama photograph. Apractical way for retrieval may be by extracting a characteristic regionfrom at least one image out of two or more images and judging whether ornot the other image has a region similar to that characteristic region.To achieve this object, the image data acquiring unit acquires the firstand second image data, and the edge pixel detecting unit and thecharacteristic region extracting unit act on either or both of theseimages.

The characteristic region defining unit acts on the first image so as toextract the characteristic region in the first image. Then, it comparesthe pixels in the characteristic region with the pixels in the secondimage so as to extract from the second image the region which coincideswith the characteristic region in the first image. Since it is expectedthat the image of the extracted region is approximately identical withthe image of the characteristic region, comparison accomplished in thisway makes it possible to extract those parts which can be superposedeasily and certainly from two or more images containing superposableparts (such as identical objects).

It is only necessary for the edge pixel extracting unit and thecharacteristic point extracting unit to be able to extract thecharacteristic region from the first image and to extract the partcoinciding with the characteristic region from the second image.Extracting the characteristic region from the first image implies thatthe object on which these units act is the first image. Of course, theseunits may act on the second image when it is necessary to reference thecharacteristic point in the second image at the time of referencing orwhen it is necessary to use the characteristic region in the secondimage.

It is only necessary for the region comparing unit to be able to comparethe pixels in the characteristic region with the pixels in the secondimage. In other words, what is needed is to able to extract a region inthe second image which is composed of pixels resembling pixels in thecharacteristic region by comparing these pixels with each other. Thisobject may be achieved in various ways. For example, comparison ofpixels will suffice if it is able to judge whether or not pixels in thefirst image are similar to pixels in the second image. Such judgment maybe accomplished by comparing pixels in terms of gray level, because asmall difference in gray level suggests similarity between the twoimages.

Various indexes may be used to judge that a difference in gray level issmall. For example, if it is desirable to extract from the second imagetwo or more region candidates coinciding with the characteristic regionin the first image, then this object is achieved by judging whether ornot the difference in gray level is lower than a prescribed thresholdvalue. If it is desirable to extract from the second image a regionwhich is most similar to the characteristic region in the first image,then this object is achieved by extracting the region in which thedifference in gray level is minimal. Of course, gray level of differentkind can be used to compare pixels. For example, in the case where thecolor of each pixel is represented by the gray level of each colorcomponent, then it is possible to use the gray level of each colorcomponent or the gray level indicating the color value (luminance, colorsaturation, and hue) of each pixel.

Practical comparison may be accomplished by extracting a region forcomparison which has the same size as the characteristic region in thesecond image and comparing pixels in the region for comparison withpixels in the characteristic region. Here, “region of the same size”means that the region has approximately the same number of pixels. Ofcourse, it is necessary that the two images have the same scale so thatthey are to be compared with each other. Two or more photographs takenunder the same condition do not need enlargement or reduction.Otherwise, at least one of photographs differing in scale should beenlarged and reduced.

Assuming that the characteristic region and the region for comparisonare of the same size, it will be possible to evaluate similarity betweenthem by computing the difference in gray level between each pixel in theformer and each pixel in the latter, which are at the correspondingposition. Therefore, if a value is obtained by summing up thesedifferences, then the resulting value permits object evaluation ofsimilarity between the two regions. This value is referred as thecomparison value. The smaller is the comparison value, the more similarare the two regions to each other.

The region comparing unit may accomplish comparison by referencing thecharacteristic point. In this case, it compares pixels in onecharacteristic point with pixels in another characteristic pixels,thereby extracting the region coinciding with the characteristic region.Assuming that the pixels in the characteristic region constitute thecharacteristic point, it extracts the pixels and their neighboringpixels in the region for comparison which correspond to the position ofthe characteristic point. If these pixels constitute the characteristicpoint, the difference in gray level between this characteristic pointand the characteristic point in the characteristic region is added tothe comparison value mentioned above.

If the pixels in the characteristic region constitute the characteristicpoint, the region comparing unit extracts the pixels and theirneighboring pixels corresponding to the position of the characteristicpoint in the region for comparison and judges whether or no these pixelsconstitute the characteristic point. If judgment is affirmative, then itcomputes the difference in gray level between the pixels constitutingthe characteristic point. Assuming that the smaller the comparisonvalue, the higher the similarity between regions, it is possible tocompute the comparison value responding to the result of comparisonbetween the characteristic points by adding the magnitude of thedifference between the characteristic points.

Incidentally, the summing up of the comparison values may beaccomplished in such a way that no substantial contribution is made fromother points than the characteristic point (which is achieved byomitting the summing up or by changing the comparison value except forthat of the characteristic point into a large value, such as the maximumvalue of difference between gray levels), then it is possible to performcomparison based on the characteristic point alone. Also, by extractingnot only the pixels corresponding to the position of the characteristicpoint in the region for comparison but also their neighboring pixels, itis possible to improve the reliability of the result of comparison. Thisis explained below. In the case where two different images are to bestitched together, the gray level is approximately the same for pixelsat the corresponding position in the regions to be superposed. However,there will be an instance in which the correspondence of the position ofpixels is not complete and there is a slight dislocation in terms ofpixel unit.

For example, in the case of superposing pixels one on top of the other,there occurs relative displacement of pixels in two photographs of thesame object on account of various effect such as lens aberration in thephotographing apparatus. By extracting not only the pixels correspondingto the position of the characteristic point in the region for comparisonbut also their neighboring pixels, it is possible to certainly extractand compare the characteristic point in the region for comparison whichcorrespond to the characteristic point in the characteristic region.Incidentally, extraction of the neighboring pixels of the pixelscorresponding to the position of the characteristic point may beaccomplished by previously establishing the pixels to be extracted(which may be, for example, 3×3 pixels, with their center being thepixel corresponding to the position of the characteristic point).

As mentioned above, the edge pixel extracting unit and thecharacteristic point extracting unit should act on the second image inorder to reference the characteristic point in the second image. If theregion comparing unit can extract the region in the second image thatcoincides with the characteristic region, then it is able to stitchtogether the first and second images at correct positions by superposingthe referenced regions.

The foregoing procedure may be modified as follows. The edge pixelextracting unit and the characteristic point extracting unit act on thesecond image and the characteristic point defining unit extracts thecharacteristic region also in the second image, so that it performscomparison based on the characteristic region extracted from both thefirst and second images. The region comparing unit compares the pixelsof the extracted characteristic region in the first image with thepixels of the extracted regions in the second image and the pixels inthe neighboring regions. In other words, since the superposing regionsof the first and second images contain pixels which are almost identicalin gray level, the characteristic regions which have been extracted byapplication of the same algorithm to the first and second images couldpossibly be the superposing regions.

Comparison that is performed on the extended region around thecharacteristic region extracted from the second image makes it possibleto certainly extract the region in the second image which coincides withthe characteristic region of the first image. Comparison in this manneris performed on the characteristic region extracted from the secondimage and its neighboring regions but not on the whole of the secondimage. It effectively limits the region to be compared, eliminatesunnecessary comparison, and helps complete the processing of comparisonrapidly. Of course, the above-mentioned procedure may be modified suchthat the neighboring regions of the characteristic region are restrictedto those within N pixels (N=natural number) around the characteristicregion.

Another possible modification may be such that the region comparing unitextracts the region in the second image which coincides with thespecific part of the first image (or the characteristic region in thesecond image) without extracting the characteristic region from thefirst image or performing the comparing process on each region. Thismodification may be practiced in such a way that the edge extractingunit and the characteristic point extracting unit act on the first andsecond images to extract the characteristic point.

That is, they create the arrangement pattern data indicating thearrangement pattern of the characteristic points in the first image andextract, based on this arrangement pattern data, the characteristicpoint approximately coinciding with the arrangement pattern from thesecond image, so that they can specify the parts of the first and secondimages which are to be superposed one on top of the other. There aremany possible ways of judging whether or not the arrangement of thecharacteristic point in the second image approximately coincides withthe arrangement pattern.

For example, they create the arrangement pattern according to the datathat specify the relative position of the characteristic point andextract the characteristic points in the second image one by one andthen judge whether or not the characteristic point exists at the sameposition as the relative position indicated by the arrangement patterndata with respect to each characteristic point. Of course, it ispermissible to judge the presence or absence of not only the relativeposition indicated by the arrangement pattern in the second image butalso its neighboring position.

There is also another possible modification, in which they not onlyjudge the presence or absence of the characteristic point as indicatedby the arrangement pattern in the second image but also compare the graylevel of the pixel if the characteristic point exists. In any case, theabove-mentioned modification makes it possible to extract the coincidingpart in the first and second images without having to extract thecharacteristic region in the first image or perform comparison for eachcharacteristic region. Needless to say, the characteristic region to beextracted from the second image is not limited to rectangular ones butit may be a region of any shape containing the characteristic point.

Incidentally, the characteristic region extracting device mentionedabove may be used alone or as a built-in component in a certain device.The present invention may be embodied variously. The above-mentionedmethod for defining the charactering region according to thedistribution of the extracted characteristic points is apparently basedon the present invention. Therefore, the present invention may beembodied also as a method, as in Claims 16 and 18. The characteristicregion extracting device may need a prescribed program to run it. Inthis case the present invention may be embodied as a program, as inClaims 17 and 19.

The concepts mentioned in Claims 2 to 15 may be applied to theabove-mentioned method and program, as a matter of course. Any storagemedium may be used to present the program; it includes, for example,magnetic recording media, magneto-optical recording media, and anyrecording media which will be developed in the future. The presentinvention may be embodied partly in the form software and partly in theform of hardware. The software may be partly recorded in a recordingmedium and read out when in need. In addition, the software may be inthe form of primary or secondary duplicate.

Other and further objects, features and advantages of the invention willappear more fully from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the computer system;

FIG. 2 is a block diagram showing the functions of the program toextract a characteristic region;

FIG. 3 is a diagram showing some examples of the pattern filter;

FIG. 4 is a process flowchart of the program to extract a characteristicregion;

FIG. 5 is a diagram illustrating an example of operation;

FIG. 6 is a block diagram showing the functions of the characteristicregion extracting/stitching program;

FIG. 7 is a process flowchart of the characteristic region stitchingprogram;

FIG. 8 a flowchart showing the comparing/joining process;

FIG. 9 is a diagram illustrating an example of operation;

FIG. 10 is a diagram illustrating an example of operation in anotherembodiment;

FIG. 11 is a diagram illustrating an example of operation in anotherembodiment; and

FIG. 12 is a flowchart showing the comparing/joining process in anotherembodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of the present invention will be described below in thefollowing order.

(1) Constitution of the system hardware

(2) Constitution and processing of the characteristic region extractingprogram

(3) Example of the characteristic region extracting process

(4) Other embodiments

(5) Process for image stitching

(1) Constitution of the System Hardware

FIG. 1 is a block diagram showing the computer system to execute thecharacteristic region extracting program according to one embodiment ofthe present invention. The computer system 10 has a scanner 11 a, adigital still camera 11 b, and a video camera 11 c, which are connectedto the computer proper 12 so that they serve as image input devices.Each image input device generates image data representing an image withpixels arranged in a dot-matrix pattern, and it outputs the image datato the computer proper 12. The image data express each of the threeprimary colors (RGB) in 256 gray levels, so that they can express about16,700,000 colors in total.

To the computer proper 12 are connected a flexible disk drive 13 a, ahard disk 13 b, and a CD-ROM drive 13 c, which serve as externalauxiliary storage devices. The hard disk 13 b stores the main programfor the system, and it reads programs and image data from the flexibledisk 13 a 1 and the CD-ROM 13 c 1 when necessary.

To the computer proper 12 is also connected a modem 14 a (as acommunication device) for connection to the external network, so that itreceives (by downloading) software and image data through the publiccommunication circuit connected to the external network. This embodimentis designed such that access to outside is achieved through the modem 14a and the telephone line. The embodiment may be modified such thataccess to the network is achieved through a LAN adaptor or access to theexternal line is achieved through a router. In addition, the computerproper 12 has a keyboard 15 a and a mouse 15 b connected thereto for itsoperation.

The computer system 10 also includes a display 17 a and a color printer17 b, which serve as image output devices. The display 17 a has adisplay area consisting of 1024 pixels (in the horizontal direction) and768 pixels (in the vertical direction). Each pixel is capable ofdisplaying 16,700,000 colors as mentioned above. This resolution is amere example; it may be changed to 640×480 pixels or 800×600 pixels.

The computer proper 12 executes prescribed programs to acquire imagesthrough the image input devices and display them on (or send them to)the image output devices. The programs include the operating system (OS)12 a as the basic program. The operating system 12 a includes thedisplay driver (DSPDRV) 12 b (for displaying on the display 17 a) andthe printer driver (PRTDRV) 12 c (for printing on the color printer 17b). These drivers 12 b and 12 c are dependent on the type of the display17 a and the color printer 17 b. They may be added to or modified in theoperating system 12 a according to the type of equipment, so that thesystem performs additional functions (other than standard ones) inherentin specific equipment used. In other words, the system performs not onlythe standard processing defined by the operating system 12 a but also avariety of additional functions available in the scope of the operatingsystem 12 a.

For execution of these programs, the computer proper 12 is equipped withCPU 12 e, ROM 12 f, RAM 12 g, and I/O 12 h. The CPU 12 e for arithmeticoperation executes the basic programs written in the RAM 12 g whileusing the ROM 12 f as a temporary work area, a setting memory area, or aprogram area. It also controls the internal and external devicesconnected through the I/O 12 h.

The application (APL) 12 d is executed on the operating system 12 a asthe basic program. It performs a variety of processes, such asmonitoring the keyboard 15 a and mouse 15 b (as the devices foroperation), controlling external devices and executing arithmeticoperation in response to operation, and displaying the result ofprocessing on the display 17 a or sending the result of processing tothe color printer 17 b.

The color printer 17 b prints characters and images with color dots onprinting paper in response to the printer driver 12 c according to printdata produced by the application 12 d. The characteristic regionextracting program of the present invention may be available as theabove-mentioned application 12 d, as the printer driver 12 c, as ascanner driver, or as a program to execute part of the functions of theapplication 12 d. It should preferably be incorporated into a program tomake panorama photographs or a program to retrieve images.

(2) Constitution and Processing of the Characteristic Region ExtractingProgram

FIG. 2 is a block diagram showing the functions of the program toextract characteristic regions. It also shows data used in theprocessing. FIG. 3 is a diagram showing some examples of the patternfilter. FIG. 4 is a process flowchart of the program to extractcharacteristic regions. The functions and processing of the program willbe explained with reference to these figures. As shown in FIG. 2, thecharacteristic region extracting program 20 consists of an image dataacquiring module 21, an edge pixel detecting module 22, a characteristicpoint extracting module 23, and a characteristic region defining module24.

It is assumed in FIG. 2 that the image data from which a characteristicregion is to be extracted is stored in the hard disk 13 b; however, theimage data may be the one which is stored in any other media or acquiredfrom the digital still camera 11 b or through the modem 14 a. Thecharacteristic region extracting program 20 works as follows. In StepS100, the image data acquiring module 21 reads image data (12 g 1) fromthe hard disk 13 b and stores it temporarily in the RAM 12 g.

In Step S105, the edge pixel detecting module 22 applies a prescribededge detecting filter to the individual pixels of the image data 12 g 1.The process in this step computes the edge gradient of the pixels, andthe edge gradient data 12 g 2 thus computed is stored in the RAM 12 g.The edge gradient can be computed by using a variety of edge detectingfilters. The data obtained after filter application may be processed invarious ways, including normalization. Whatever filter may be used, theedge gradient data 12 g 2 should have gray levels indicating the edgegradient at each pixel.

In Step S110, the edge pixel detecting module 22 determines whether ornot each pixel is an edge by referencing the edge gradient data 12 g 2which has been computed by the edge pixel detecting module 22 asmentioned above. In other words, the edge gradient of each pixel iscompared with a previously established threshold value and the pixelwhose edge gradient is larger than the threshold value is defined as theedge pixel. After the edge pixel has been defined, the edge detectingdata 12 g 3 (which indicates whether or not each pixel is an edge pixel)is stored in the RAM 12 g. The edge detecting data 12 g 3 representseach pixel with 2 bits, that is, “1” denotes that each pixel is an edgepixel and “0” denotes that each pixel is not an edge pixel.

In Step S115, a judgment is made as to whether or not the processing todetermine whether or not pixels are edge pixels has been performed onall pixels in the image data 12 g 1 acquired as mentioned above. Theprocedure in Steps S105 and S110 is repeated until a judgment is made inStep S115 that the processing has been performed on all pixels. As theresult, the edge gradient data 12 g 2 indicate the edge gradient for allpixels in the image data 12 g 1 and the edge detecting data 12 g 3indicates whether or not pixels are edge pixels for all pixels in theimage data 12 g 1.

In Steps S120 to S140, the characteristic point extracting module 23applies the pattern filter data 12 g 4 to the edge detecting data 12 g3, thereby extracting characteristic points from the edge pixels, andthen stores the characteristic point data 12 g 5 (which represent thecharacteristic points) in the RAM 12 g. The pattern filter data 12 g 4is the filter data which represent a pattern formed by edge pixels andtheir neighboring pixels. It represents each of the 3×3 pixels by “1” or“0”.

FIG. 3 shows some filters which the pattern filter data 12 g 4 (storedin the RAM 12 g) represent in this embodiment. In these pattern filters,“1” represents the edge pixel and “0” represents the non-edge pixel. Allof them have “1” at the center and consecutive four 1's or 0's in theneighboring eight pixels. Moreover, the pattern filter data 12 g 4 haseight sets of data for filters differing in the position of “1” or “0”.

Each of the pattern filters shown in FIG. 3 has 1's and 0's arranged asfollows. The middle line between the line of three 1's and the line ofthree 0's contains horizontally or vertically consecutive two 1's, andthe diagonal line contains consecutive two 1's. Therefore, the linesegments passing through each boundary between “1” and “0” form an angleof 135°. Thus, the 3×3 pattern filter in this embodiment extracts anangle of 135°.

In Step S120, the characteristic point extracting module 23 referencesthe edge detecting data 12 g 3 and applies the pattern filter data 12 g4 to the edge pixel. In other words, the edge pixel is superposed on thecentral pixel of the pattern filter shown in FIG. 3. In Step S125, thecharacteristic point extracting module 23 compares the edge detectingdata 12 g 3 with the pattern filter data 12 g 4 in the eight neighboringpixels around the central pixel, so that it judges whether or not bothsets of data have two or more pixels representing “1” at the sameposition. In other words, it judges whether or not the pattern filter issimilar to the edge detecting data 12 g 3.

Since the pattern filter data 12 g 4 is applied to edge pixels, if boththe edge detecting data 12 g 3 and the pattern filter data 12 b 4 haveneighboring pixels two or more of which represent “1”, then coincidenceof pixel “1” should occur at three or more positions. In thisembodiment, it is determined under this condition that the periphery ofthe edge pixel in the edge detecting data 12 g 3 resembles the patternformed by the pattern filter. Therefore, the above-mentioned conditionmay be used to judge whether or not the edge formed by the edgedetecting data 12 g 3 contains a pattern similar to an angle of 135°.

If it is judged in Step S125 that there are two or more coincidences of“1” in the periphery of the edge pixel, this edge pixel is registered asthe characteristic point. The pixel which has been registered as thecharacteristic point functions as a flag “1” indicating that theregistered pixel is the characteristic point. This data is stored as thecharacteristic point data 12 g 5 in the RAM 12 g. The characteristicdata 12 g 5 is only required to have a set of data indicating theposition of the characteristic point and also indicating that the datarepresents the characteristic point. It may be a set of dot-matrix data,with “1” representing the characteristic point and “0” representing thenon-characteristic point. Alternatively, it may be a set of data whichindicates the coordinate of the characteristic point.

In Step S135, a judgment is made as to whether or not the pattern filterhas been applied to all edge pixels in the edge detecting data 12 g 3.Until an affirmative result is obtained, Steps S120 to S130 are repeatedfor each pattern filter. Further, in Step S140, a judgment is made as towhether or not Steps S120 to S135 have been performed on all the patternfilters shown in FIG. 3. Until an affirmative result is obtained, StepsS120 to S135 are repeated. In this way there is obtained thecharacteristic point data 12 g 5 in which the edge pixels forming anangle of 135° represent the characteristic point which has beenextracted.

In Steps S145 to S160, the characteristic region defining module 24references the characteristic point data 12 g 5 and the edge gradientdata 12 g 2, thereby extracting the characteristic region. This isaccomplished as follows. In Step S145, it divides the image into severalregions of prescribed size. In Step S150, it references thecharacteristic point data 12 b 5 and counts the number of characteristicpoints for each of the divided regions, and it extracts any region whichhas more than α characteristic points. (α is a previously establishedthreshold value.)

In other words, it limits the number of regions having manycharacteristic points according to the discrimination based on thethreshold value α, thereby narrowing down the candidates forcharacteristic regions. In Step S155, it further references the edgegradient data 12 g 2 and sums up the values of the edge gradient ofpixels existing in each region which has been extracted in Step S150,thereby computing their average value. In Step S160, it defines theregion having the average value larger than β as the characteristicregion, and it stores in the RAM 12 g the characteristic region data 12g 6 which indicates the position of the characteristic region in theimage. Here, β is a previously established threshold value.

(3) Example of the Characteristic Region Extracting Process

The following illustrates by example how the above-mentionedconstitution and processing work. FIG. 5 is a diagram illustratingextraction of the characteristic region from the image data 12 g 1 of aphotograph of mountains. The edge pixel detecting module 2 applies theedge detecting filter, thereby creating the edge gradient data 12 g 2and the edge detecting data 12 g 3. Incidentally, the edge detectingfilter shown in FIG. 5 is a Sobel filter; however, it may be replaced byany other filter.

Although both the edge gradient data 12 g 2 and the edge detecting data12 g 3 represent the edge detected from the image data 12 g 1, theformer has the gray level indicating the edge gradient for each pixel(of eight-bit data). As shown in FIG. 5, the edge pixel may have a graylevel as large as 230 and the non-edge pixel may have a gray level assmall as 0.

The characteristic point extracting module 23 references the edgedetecting data 12 g 3 and applies the pattern filter data 12 g 4 to theedge pixel. This operation creates the characteristic point data 12 g 5,which is a set of dot-matrix data composed of “1” and “0” as shown inFIG. 5, with “1” representing the edge pixel whose neighboring pixelsare similar to the pattern represented by the pattern filter, and “0”representing pixels excluding the characteristic points. Incidentally,in the characteristic point data 12 g 5, black points represent thecharacteristic points and broken lines represent the positioncorresponding to the edge. The foregoing procedure narrows down thecandidates of the characteristic regions selected from the image data.In other words, the characteristic and differentiable part in the imageis confined to the part where the characteristic point exists. So, it isonly necessary to find out the really characteristic part from theconfined part, and this eliminates the necessity of repeating thesearching procedure many times.

The characteristic region defining module 24 divides the image into asmany regions as necessary and extracts one region having more than αcharacteristic points, as shown at the bottom of FIG. 5, in which brokenlines represent the divided regions and solid lines represent theregions to be extracted. In the example shown, the regions to beextracted have more than two characteristic points. In this embodiment,the characteristic region defining module 24 computes the average valueof the edge gradient in each of the extracted regions, and it definesthe region having an average value larger than β as the characteristicregion. Incidentally, in FIG. 5, the region to be extracted as thecharacteristic region is indicated by thick solid lines.

As mentioned above, according to this embodiment, the procedure that isperformed on all pixels is limited to edge detection, which is a simpleoperation with a filter consisting of a few pixels. Other procedures areperformed on a very small portion of the image, such as edge pixels andcharacteristic points. Operation in this manner is very fast.

(4) Other Embodiments

The foregoing merely exemplifies the constitution and processing forextraction of the characteristic regions but does not exclude possiblemodifications. For example, although it is assumed in FIG. 5 that alldata are pixel data arranged in a dot-matrix pattern, they may bereplaced by data representing detected edges or coordinate forcharacteristic points. Such data save the amount of data. Any other dataformat may be adopted.

The foregoing embodiment, in which extraction of the characteristicregion is accomplished by diving the image and counting the number ofcharacteristic points in each of the divided regions, may be so modifiedas to extract those parts having many characteristic points from thecharacteristic point data 12 g 5 and define the extracted part as thecharacteristic region if it has a prescribed size. Moreover, theforegoing embodiment, in which the region having a large average valueof edge gradient is defined as the characteristic region, may bemodified such that the region having many characteristic points isdefined as the characteristic region.

The foregoing embodiment, in which a threshold value of α is used tojudge whether or not there are many characteristic points and athreshold value of β is used to judge whether or not the average valueof edge gradient is large, may be so modified as to employ any otherthreshold values. In addition, judgment by means of the threshold valuemay be replaced by judgment by means of the number of characteristicpoints or the maximum average value of edge gradients or the upper threevalues of edge gradients. The former procedure permits extraction of onemost characteristic point and the latter procedure permits extraction ofmore than one characteristic point.

In addition, the procedure of applying the pattering filter to the edgepixels and judging whether or not the prescribed pattern to be extractedis formed may also be variously modified. For example, it may bemodified such that the value of the pattern filter to be compared is notonly “1” but also “0”. Modification in this manner adjusts the accuracyof extraction of the desired object.

The present invention may be used in various situations. For example,the characteristic region extracting program of the present inventionmay be applied to two images to be stitched together to make a panoramaphotograph. In this case, it is necessary to specify two regions (oneeach from two images) which coincide with each other. The programfacilitates extracting such regions and specifying the position forstitching.

It will be possible to improve the accuracy with which the coincidingregions are specified, if the above-mentioned threshold value isadjusted such that more than one characteristic region are extracted orby imposing the condition that the characteristic region at the leftside of one image should coincide with the one at the right side ofanother image. Another condition may be established by comparing thecharacteristic regions selected from the two images in terms of thenumber of characteristic points or the average value of edge gradientand then defining two regions having the most similar values as thecoinciding regions.

The present invention may also be applied to image retrieval. Retrievingan image containing a part coinciding with or similar to a part of anoriginal image is accomplished by counting the number of characteristicpoints in the original image and computing the average value of edgegradient in the original image and then extracting the characteristicregion for the image to be retrieved. The thus obtained characteristicregion also undergoes the counting of the characteristic points and thecomputation of the average value of edge gradient. Then thecharacteristic region is compared with the original image in terms ofthe number of characteristic points and the average value of edgegradient. If there exists a characteristic region, in which the numberof characteristic points and the average value of edge gradientcoincide, then the image containing such a characteristic region may beregarded as the retrieved image. If there exists a characteristicregion, in which these values are similar, then the image containingsuch a characteristic region may be regarded as the similar image.

(5) Process for Image Stitching

The following is concerned with the procedure for stitching two or moreimages together. FIG. 6 is a block diagram showing the functions of thecharacteristic region extracting/stitching program 200 which is intendedto extract the characteristic regions and then stitching two imagestogether. It also shows the data used in the execution of the program.FIG. 7 is a process flowchart of the characteristic regionextracting/stitching program. Reference to these figures will be made infollowing description about the function and processing of the program.Incidentally, the characteristic extracting/stitching program 200 hasmany in common with the characteristic region extracting program 20mentioned above. Such common parts in the constitution and procedure areindicated by the same symbols as used in FIGS. 2 and 4, and theirexplanation is partly omitted from the following description.

The characteristic region extracting/stitching program 200 is comprisedof the image data acquiring module 21, the edge pixel detecting module22, the characteristic point extracting module 23, and thecharacteristic region defining module 24, as in the case of FIG. 2. Itis further comprised of the region comparing module 250 and the imagestitching module 260. For image stitching, the image data 130 a and theimage data 130 b are stored in the hard disk 13 b. The imagesrepresented by the image data 130 a and 130 b will be referred to asimage A and image B, respectively. Image A and image B in thisembodiment correspond respectively to the first image and the secondimage in the embodiment mentioned above.

The image data acquiring module 21, the edge pixel detecting module 22,the characteristic point extracting module 23, and the characteristicregion defining module 24 perform the same processing as those of thecharacteristic region extracting program 20 shown in FIG. 2. The onlydifference is that their processing is performed on two sets of imagedata. In other words, the image data acquiring module 21 works in StepS100′ to read the image data 130 a and 130 b from the hard disk 13 b andtemporarily store them in the RAM 12 g. (The stored image data areindicated by 121 a and 12 ab.)

Thus, there are two images to be processed. The edge detecting module 22and the characteristic point extracting module 23 processes the imagedata 121 a and 121 b in Steps S105 to S142. The processing in Steps S105to S140 is identical with that shown in FIG. 4. That is, in the firstloop from Step S105 to S142, the edge pixel detecting module 22 detectsthe edge gradient of the image A according to the image data 121 a anddefines the detected result as the edge gradient data 122 a and the edgedetecting data 123 a, and the characteristic point extracting module 23extracts the characteristic point according to the edge detecting data123 a and the pattern filter data 12 g 4 and defines the extractedresults as the characteristic point data 125 a.

After the processing according to the image data 121 a has beencompleted, the program judges in Step S142 that the processing of theimage data 121 b (image B) is not yet completed. Then, the programswitches the object of processing to the image data 121 b in Step S144and repeats the process in Step S105. As the result, the edge pixeldetecting module 22 creates the edge gradient data 122 b and the edgedetecting data 123 b, and the characteristic point extracting module 23creates the characteristic point data 125 b.

In this embodiment, the program extracts the characteristic region onlyfrom the image A and searches for the region coinciding with thischaracteristic region in the image B. Therefore, the processing in StepsS145′ to S160′ is performed on the image A. In other words, in StepS145′, the characteristic defining module 24 divides the image intoseveral regions of prescribe size. In Step S150, it references thecharacteristic point data 125 a mentioned above, thereby counting thenumber of characteristic points in each of the divided regions, andextracts the regions in which there are more than a characteristicpoints. In Step S155, it references the edge gradient data 122 amentioned above, thereby summing up the edge gradient of pixels existingin each of the regions extracted in Step S150, and it computes theiraverage value.

In Step S160′, the program defines the region in which the average valueis larger than β as the characteristic region SA in the image A, and itstores in the RAM 12 g the characteristic region data 126 a whichindicates the coordinate (SA(X,Y)) of the characteristic region in theimage A. Incidentally, in this embodiment, no upper limits are imposedon the number of characteristic points extracted from the characteristicpoint data 125 a and the threshold value α mentioned above; however, thenumber of characteristic points to be extracted may be adjusted to about200 to 300 and the threshold value α may be adjusted to about 10 to 30when natural images are stitched together. The former adjustment may beaccomplished by adjusting the threshold value for adjustment on the edgepixel in Step S110. The coordinate SA(X,Y) is not specificallyrestricted so long as they specify four corners of a rectangle ofprescribed size. Such a coordinate may be replaced by those whichspecify the diagonal line of the rectangle.

After completion of the characteristic region data 126 a which indicatesthe characteristic region in the image A, the region comparing module250 and the image stitching module 260 perform the procedure forcomparison and stitching on the image A and image B, in Step S165. FIG.8 is a flowchart showing the comparing/stitching process. The regioncomparing module 250 references the characteristic region data 126 a andthe characteristic point data 125 a and 125 b, thereby extracting theregion, which coincides with the characteristic region SA, from theimage B. It carries out the procedure in Steps S200 to S280.

In Step S200, the region comparing module 250 establishes the coordinateSB(X,Y) of the position of the region candidate as a variable so as toestablish the region of the same size as the characteristic region SA asthe region candidate SB in the image B, and initializes the coordinateto (0,0). It also establishes the minimum comparing value variable M0 tosubstitute the minimum value for comparison between the pixel in thecharacteristic region SA and the pixel in the region candidate SB, andthen it initializes by substituting the maximum value that can be takenas the comparison value. Here, the comparison value is a value obtainedby summing up the differences of the gray levels of individual pixels inthe region. If the gray level is represented with 8 bits (in which casethe gray level ranges from 0 to 255), the maximum comparing value is 256multiplied by the number of pixels in the region.

In Step S205, the region comparing module 250 establishes the coordinateoffset (I,J) as a variable to indicate the coordinate to specify theposition of the pixel in the characteristic region SA and the positionof the pixel in the region candidate SB, and then it initializes thevalue to (0,0). It also establishes the comparing value variable M tosubstitute the comparing value to be computed for each region, and itinitializes the value to 0. In Step S210, it establishes the differencevalue variable V to substitute the difference value for each pixel whenthe pixel is compared, and it substitutes the maximum value to be takenas the difference value and initializes it. This difference value is adifference of the gray level of the pixel, and the maximum comparingvalue is 256 if the gray level is represented with 8 bits.

In this embodiment, the pixel in the region candidate SB to be comparedwith the specific pixel in the characteristic region SA is notrestricted to the pixel at the position corresponding to the specificpixel but the neighboring pixels are also compared. So, in Step S210,the module establishes the neighboring pixel variable (K,L) to specifythe neighboring pixels, and then it initializes the value to (0,0). InStep S215, the module references the characteristic point data 125 a(which indicates the characteristic point of the image A) and thecharacteristic region data 126 a, thereby judging whether or not thecoordinate SA(X+I, Y+J) is the characteristic point.

If the module does not judge, in Step S215, that the coordinate SA(X+I,Y+J) is not the characteristic point, then it proceeds to Step S250 toupdate the comparing value variable M with the value which is obtainedby adding the difference value variable V to the comparing valuevariable M. In other words, the module computes comparing value in sucha way that the difference of the gray level of the pixel is not computedfor the pixel which is not the characteristic point but it is added tothe maximum value of the difference value variable V. If the modulejudges, in Step S215, that the coordinate SA(X+I, Y+J) is thecharacteristic point, then it proceeds to Step S220 to reference thecharacteristic point data 125 b indicating the characteristic point ofthe image B, thereby judging whether or not the coordinate SA(X+I+K,Y+J+K) is the characteristic point.

If the module does not judge, in Step S220, that the coordinateSA(X+I+K, Y+J+K) is the characteristic point, then it proceeds to StepS240 in which it substitutes any of “−1, 0, 1” into the respectivevariables K and L and judges whether or not processing has beenperformed on all of the combinations. If the module does not judge, inStep S240, that the processing has been performed on all of thecombinations, then it proceeds to Step S245 in which it changes thevalue of (K,L) by any of “−1, 0, 1” and repeats the processing afterStep S220.

In other words, the module performs processing on all of the arbitrarycombinations, with the value of (K,L) being “−1, 0, 1”. Therefore, itperforms the processing in Steps S220 to S235 on 3×3 pixels whose centeris at the coordinate SB(X+I, Y+J). In addition, since it specifies thecoordinate of the common angle (say, the upper left angle) of eachregion for any of the coordinate SA and the coordinate SB, the relativeposition of the coordinate SA(X+I, Y+J) with respect to the angle of thecharacteristic region SA coincides with the relative position of thecoordinate SB(X+I, Y+J) with respect to the angle of the regioncandidate. Therefore, in this embodiment, the module makes judgment onnot only the coordinate in the region candidate SB with respect to thecharacteristic point in the characteristic region SA but also thecharacteristic point including the pixels in the neighborhood of thecoordinate.

If the module judges, in Step S220, that the coordinate SB(X+I+K, Y+J+K)is the characteristic point, then it computes abs(PA(X+I, Y+J)−PB(X+I+K,Y+J+L)) and substitutes it into the temporary variable V0 of thedifference value. Here, abs denotes an absolute value, PA(X+I, Y+J)denotes the gray level of the pixel at the coordinate SA(X+I, Y+J), andPB(X+I+K, Y+J+L) denotes the gray level of the pixel at the coordinateSB(X+I+K, Y+J+K).

Incidentally, the gray level of each pixel may be represented by variousvalues; for example, it may be represented by the gray level for eachcolor component of each pixel, or it may be represented by color values(luminance, color saturation, and hue) for the color of each pixel. Ifthe module judges, in Step S220, that the coordinate SB(X+I+K,Y+J+L) isthe characteristic point, then it skips Steps S225 and S230 and makesjudgment in Step S240.

In Step S230, the module judges whether or not the value of thetemporary variable V0 of the difference value is smaller than thedifference value variable V of each pixel mentioned above. If the modulejudges that the value of the temporary variable V0 of the differencevalue is smaller than the difference value variable V of each pixelmentioned above, then it proceeds to Step S235 in which it substitutesthe value of the variable V0 into the variable V. If the module does notjudge, in Step S235, that the value of the temporary variable V0 of thedifference value is smaller than the difference value variable V of eachpixel mentioned above, then it skips the step S235.

After the foregoing processing, the module makes judgment in Step S240.If it judges in Step S240 that the processing has been performed on allthe combinations of (K,L) created by selecting any of “−1, 0, 1”, thenit proceeds to Step S250 in which it updates the comparing valuevariable M with the value obtained by adding the difference valuevariable V to the above-mentioned comparing value variable M. In thecase where the processing in Steps S220 to S245 has been performed, theprocessing to update the variable V when the temporary variable V0 ofthe difference value is smaller than the difference value variable V isperformed; therefore, it follows that the difference value variable Vhas been substituted by the minimum value among the difference valuesobtained by comparing with the above-mentioned 3×3 pixel.

In Step S255, the module judges whether or not the above-mentionedprocessing has been performed on all pixels in the region. In otherwords, it judges whether or not the processing has been performed byestablishing all the combinations which are obtained by using anarbitrary value up to the prescribed upper limit value of I,J. If themodule does not judge that the foregoing processing has been performedon all pixels in the region, then it proceeds to Step S260 in which itupdates (I,J) and repeats the processing after Step S210.

The above-mentioned processing is not intended to compute the differencebetween the pixel in the region candidate SB and all the pixels in thecharacteristic region SA but is intended to compute the differencebetween the characteristic point in the characteristic region SA and thecharacteristic point in the region candidate SB and add the result tothe comparing value variable M. If the characteristic point is notinvolved, the maximum value of the difference for each pixel is added tothe comparing value variable M, as mentioned above. As the result ofthis processing, contribution from pixels which are not characteristicpoints becomes the same value (maximum value). Therefore, it followsthat the smaller the value of the comparing value M which is eventuallyobtained, the smaller the difference in the gray level of thecharacteristic point.

In Step S265, the module judges whether or not the value of thecomparing value variable M is smaller than the value of the minimumcomparing value variable M0. If it judges that the value of thecomparing value variable M is smaller than the value of the minimumcomparing value variable M0, it regards the value of the variable M asthe minimum value at that time and then updates the variable M0 with thevariable M in Step S270. In this embodiment, the region candidate SB inwhich the comparing value is the minimum value is regarded as the regionwhich coincides with he characteristic region SA. So, in Step S270, themodule establishes the variable of the comparing position coordinate SB(X0, Y0) which indicates the region in the image B coinciding with thecharacteristic region SA, and it substitutes the value of the coordinateSB(X, Y) into the coordinate SB(X0, Y0).

If the module does not judge, in Step S265, that the value of thecomparing value variable M is smaller than the value of the minimumcomparing value variable M0, then it skips Step S270 by regarding theregion candidate SB as not coinciding with the characteristic region SA.In Step S275, the module judges whether or not the processing has beencompleted over the entire range of the image B. In other words, itjudges whether or not the coordinate SB(X, Y) of the region candidateposition has been established for all the coordinates which would bepossible in the image B. If the module judges, in Step S275, that theprocessing has been completed over the entire range of the image B, itproceeds to Step S280 in which it updates the coordinate SB(X,Y) andrepeat the process after Step S205.

In the case where judgment in Step S275 shows that the processing hasbeen completed over the entire range of the image B, the coordinate ofthe region which has become the minimum comparing value M as the resultof comparison is SB(X0, Y0). Therefore, the region in the image Bcoinciding with the characteristic region SA is specified by thecoordinate SB(X0, Y0) of its angle.

So, in Step S285, the image stitching module 260 receives the coordinateSB(X0, Y0), references the characteristic region data 126 a, therebyacquiring the coordinate SA(X, Y), stitches the image data 121 a and 12ab together such that the coordinate SA(X, Y) coincides with thecoordinate SB(X0, Y0), and outputs the stitching image data. Theforegoing processing makes it possible to stitch the image A and theimage B together by accurately superposing those parts having the samecontent.

The following illustrates by example how the above-mentionedconstitution and processing work. FIG. 9 is a diagram illustrating howto stitch the image A and the image B together at an appropriateposition, both images being photographs of mountains. The image A iscomposed of a large mountain in the foreground and a summit of anothermountain in the background. The image B is composed of the right side ofa mountain in the foreground and the right side of another mountain inthe background.

When the edge detecting module 22 and the characteristic pointextracting module 23 processes the image data 121 a and 121 brepresenting these images, they detect several characteristic pointsalong the ridgeline of the mountains. These characteristic points arerepresented by the characteristic point data 125 a and 125 b. These datarepresent the characteristic point with “1” and the non-characteristicdata with “0” for pixels forming a dot matrix. Therefore, the data looklike the schematic diagram shown in FIG. 9 (the second row).

The characteristic region defining module 24 extracts the characteristicregion from the image A and determines the coordinate SA(X,Y) of theupper left corner of the characteristic region SA, as shown in FIG. 9(the third row). The characteristic region defining module 24 alsoestablishes the region candidate SB for the image B. The regioncomparing module 250 computes the comparing value M while changing theposition of the region candidates SB (or scanning the region in theimage B). When this scanning is completed, the region (the coordinateSB(X0, Y0)) which has been given the minimum comparing value M0 is theregion which has the least difference between the gray levels of thepixels of the characteristic point as compared with the characteristicregion SA.

Consequently, the image of the coordinate SB(X0, Y0) coincides with theimage of the coordinate SA(X, Y), as shown in FIG. 9. Thus, the imagestitching module 260 makes it possible to stitch the images A and Btogether by superposing the coordinates, thereby accurately superposingthe two images one over another. In this embodiment, the module extractsthe characteristic points from the image A and determines thecharacteristic region of the image A according to the characteristicpoints. This procedure permits easy and rapid extraction of the parts(as the basis for superposition) from the image A.

Moreover, in this embodiment, the difference between thesecharacteristic points is computed for the image A and the image B. Thisobviates the necessity of computing the differences of all pixels forthe images A and B and of trying the matching. This leads to rapidprocessing. The advantage of extracting edges and detectingcharacteristic points is that characteristic parts can be extractedeasily and objectively from complex images and their comparison can beaccomplished for image stitching based on these characteristic parts.This permits accurate and rapid processing.

The foregoing processing is a mere example which does not restrict theprocessing to specify the positions to be joined together in two or moreimages. It may be modified such that the object to be scanned by theregion candidate SB is not the whole of the image B but a portion of theimage B. To be concrete, the construction almost identical with thatshown in FIG. 6 is feasible. However, the characteristic region definingmodule 24 extracts not only the characteristic region SA in the image Abut also the characteristic region SB′ in the image B. It is desirableto extract more than one region as the characteristic region SB′. Thismay be accomplished by adjusting the above-mentioned threshold value αto define it as the object for comparison and by defining the region, inwhich the average edge gradient has a value larger than the prescribedthreshold value, as the characteristic region SB′.

FIG. 10 is a diagram illustrating how this embodiment is applied to theimages A and B shown in FIG. 9. That is, the characteristic regiondefining module 24 creates the characteristic point data 125 a and 125b, as in the constitution shown in FIG. 6, and extracts thecharacteristic region SA and the characteristic region SB′. Comparisonis accomplished by the procedure almost similar to that shown in FIG. 8;however, the object to be scanned as the region candidate SB is thecharacteristic region SB′ and its neighboring regions. For example, theobject for scanning is the area indicated by the broken line whichsurrounds the characteristic region SB′ as shown in FIG. 10. In StepS280, comparison is accomplished by changing the coordinate in such arestricted region.

It is expected that if the algorithm to extract the characteristicregion from the image A is employed, it will be possible to extract fromthe image B the region having the same feature as that characteristicregion. Thus, if the same algorithm as used for the image A is appliedalso to the image B and the characteristic region is extracted from theimage B, it will not be necessary to assign the image B entirely to theregion candidate and hence it will be possible to complete theprocessing very fast.

In the case shown in FIG. 10, the neighbor of the characteristic regionSB′ is also scanned, so that the part coinciding with the characteristicregion SA is certainly extracted. The area of the neighbor of thecharacteristic region SB′ may be properly adjusted according tocertainty required. Scanning of the neighbor may be omitted in othercases.

It is not essential to compare pixels in the region of specific shape(e.g., rectangle) as in the above-mentioned embodiment; however,modification is possible in which these parts which coincide with eachother in the images A and B are detected by using the relative positionof the characteristic points. An example of such modification applied tothe images A and B (shown in FIG. 9) is illustrated in FIG. 11, and themodified embodiment is accomplished according to the flowchart shown inFIG. 12. This modified embodiment employs the same modules and data asshown in FIG. 6.

The modules shown in FIG. 6 compute the characteristic point data 125 aand 125 b for the images A and B, respectively, and also compute thecharacteristic region SA in the image A. An example of thecharacteristic region extracted from the image A is shown in the left ofFIG. 11 (which is an enlarged view). Black dots in the enlarged viewdenote the characteristic points. In this modified embodiment, theregion comparing module works differently from the region comparingmodule shown in FIG. 6. That is, it acquires the arrangement pattern ofcharacteristic points in the characteristic region SA, thereby creatingthe arrangement pattern data, and extracts from the image B those partsconforming to this arrangement pattern.

Consequently, the region comparing module in this modified embodimentexecutes the processing shown in FIG. 12 in place of the processingshown in FIG. 8. In other words, it extracts a pixel which is acharacteristic point in the characteristic region SA and is at theleftmost side, and then it assigns this pixel to the coordinate A(X0,Y0). (Step S1650) Then it creates the arrangement pattern dataindicating the relative position of the coordinate A(X0, Y0) and othercharacteristic points. (Step S1655).

The relative position may be described by, for example, retrieving thecharacteristic points rightward from the coordinate A(X0, Y0) andspecifying the respective numbers (xa) and (yb) of horizontal andvertical pixels existing between the first found characteristic pointand the coordinate A(X0, Y0). It is possible to create the arrangementpattern by sequentially assigning (xa,ya), (xb,yb), . . . to the numberof horizontal and vertical pixels for the characteristic points whichare found during moving rightward from the coordinate A(X0, Y0).

Needless to say, the mode of the arrangement pattern data is not limitedto the one mentioned above. In any case, after the arrangement patternhas been created, it is compared with the characteristic point of theimage B according to the arrangement pattern. In other words, itextracts one characteristic point from the characteristic point data 125b and substitute its coordinate into the coordinate b(X0, Y0). (StepS1660) Then, it references the arrangement pattern data mentioned above,thereby sequentially extracting the pixels which exist at the right sideof the coordinate b(X0, Y0) and correspond to the position of thecharacteristic point in the characteristic region SA. In other words, itadds xa and ya respectively to the x coordinate and the y coordinate ofthe coordinate b(X0, Y0), thereby specifying the position of the pixel.It further adds xb and yb to these coordinates, thereby specifying theposition of these pixels. And, it acquires the gray level of each pixeland compares it with the characteristic point in the image A. In otherwords, it computes the difference between the gray level of the pixel ofeach characteristic point in the image A and the gray level of the pixelwhich corresponds to each characteristic point and which has beenextracted from the image B. It further adds up the difference values andassign the resulting sum to the comparing value M. (Step S1665)

After the comparing value M has been calculated, it judges whether ornot it is smaller than the value of the variable M0 into which theminimum comparing value is to be substituted. (Step S1670) If it judgesthat the value of the comparison value M is smaller than the value ofthe variable M0, then it regards the value of the variable M as theminimum value at that time and updates the variable M0 with the value ofthe variable M. (Step S1675) Also, at this time, it substitutes thevalue of the coordinate b(X0, Y0) into the variable B(X0, Y0) into whichthe coordinate of the image B to be supposed on the coordinate A(X0, Y0)is to be substituted. Incidentally, in the initial stage of processing,the variable M0 is previously initialized with a sufficiently largevalue.

And, in Step S1680, it judges whether or not the processing has beencompleted, with the value of all characteristic points in the image Bbeing assigned to the coordinate b(X0, Y0). It updates, in Step S1685,the coordinate b(X0, Y0) and repeats the processing after Step S1665,until it judges that the processing for all characteristic points iscompleted. The coordinate which is registered as the value of thecoordinate B(X0, Y0) after the above-mentioned repeated processing isthe coordinate in the image B, which is the coordinate to be superposedon the coordinate A(X0, Y0) of the image A. Therefore, the imagestitching module performs the processing for stitching the images A andB together while superposing the coordinate A(X0, Y0) on the coordinateB(X0, Y0).

The above-mentioned processing makes it possible to stitch the images Aand B together without forming a region of prescribed size in the imageB and performing the processing for comparison of pixels in this region.Incidentally, in this embodiment, it is only necessary to extract acertain part having some kind of feature in the arrangement pattern fromthe image A and create the arrangement pattern data, and the processingfor extracting the characteristic region from the image A is not alwaysessential. Moreover, the forgoing embodiment is not always limited tostitching two images together. It may be applied to the situation inwhich the right and left sides of one image are joined to other twoimages, or the right and left sides and upper and lower sides of oneimage are joined to other four images.

As mentioned above, the present invention will provide a computer whichrealizes the function to retrieve characteristic parts in an image andthe function to stitch two or more images together. Such functions arenecessary to retrieve a image (or part thereof) taken by a digitalcamera and search for the position for joining to make a panoramaphotograph.

The foregoing invention has been described in terms of preferredembodiments. However, those skilled, in the art will recognize that manyvariations of such embodiments exist. Such variations are intended to bewithin the scope of the present invention and the appended claims.

1. A characteristic region extracting device which comprises an imagedata acquiring unit to acquire image data representing an image withpixels arranged in a dot matrix pattern, an edge pixel detecting unit todetect the edge pixels of the image according to the image data, acharacteristic point extracting unit to extract the edge pixel as thecharacteristic point when a pattern formed by the detected edge pixelsand their neighboring pixels is similar to a prescribed object to beextracted, and a characteristic region defining unit to define as thecharacteristic region the prescribed region in the image which has theextracted characteristic points in large number.
 2. The characteristicregion extracting device according to claim 1, wherein thecharacteristic point extracting unit is a dot-matrix filter formed withthe filter value indicating the edge pixel and the filter valueindicating the non-edge pixel, so that it compares more than one filterforming a pattern by the arrangement of each filter value with the edgedetecting data after detection of the edge pixel and determines whetheror not the pixel corresponding to the center of the filter is thecharacteristic point.
 3. The characteristic region extracting deviceaccording to claim 2, wherein the filter is applied to each edge pixelat the time of comparing and the edge element to which the filter hasbeen applied is defined as the characteristic point when the filtervalue indicating the edge pixel coincides with the edge pixel of theedge detecting data at more than two places.
 4. The characteristicregion extracting device according to claim 1, wherein the pattern to beextracted is a pattern in which the edges form an angle larger 90° andsmaller than 180°.
 5. The characteristic region extracting deviceaccording to claim 2, wherein the filter is a filter of 3×3 pixels, withadjoining four pixels (excluding the center) being the filter valueindicating the edge pixel and other adjoining four pixels being a filtervalue indicating the non-edge pixels.
 6. The characteristic regionextracting device according to claim 1, wherein the characteristicregion defining unit divides the image into two or more regions having aprescribed number of pixels and defines as the characteristic region theregion in which the number of characteristic points exceeds a specificthreshold value.
 7. The characteristic region extracting deviceaccording to claim 6, wherein the characteristic region defining unitcomputes the average value of the edge gradient of pixels contained ineach the two or more regions and defines as the characteristic regionthe region in which the average value is high.
 8. The characteristicregion extracting device according to claim 1, wherein the image dataacquiring unit acquires the first image data indicating the first imageand the second image data indicating the second image, and thecharacteristic region defining unit defines the characteristic region inthe first image, and which further comprises a region comparing unit tocompare the pixel in the characteristic region with the pixel in thesecond image.
 9. The characteristic region extracting device accordingto claim 8, wherein the region comparing unit compares the gray level ofthe pixel in the characteristic region with the gray level of the pixelin the second image and extracts the region in the second image in whichthe difference between two is small.
 10. The characteristic regionextracting device according to claim 9, wherein the region comparingunit extracts the region for comparison which has the same size as thecharacteristic region in the second image and adds up the differencesbetween the gray level of the pixel in the extracted region forcomparison and the gray level of the pixel in the characteristic region,thereby defining the resulting summed value as the comparing value ofthe extracted region, and extracts the region in which the comparingvalue is small.
 11. The characteristic region extracting deviceaccording to claim 10, wherein, when the pixel in the characteristicregion is the characteristic point, the region comparing unit extractsthe pixel and its neighboring pixels which corresponds to the positionof the characteristic point in the region for comparison, and, whenthese pixels are the characteristic points, adds the magnitude ofdifference between the gray level thereof and the gray level of thecharacteristic point in the characteristic region to the comparingvalue.
 12. The characteristic region extracting device according toclaim 8, which further comprises a stitched image data creating unit tocreate the stitched image data which represents the stitched imageformed by stitching together the first image and the second image bysuperposing the region in the second image which has been extracted bycomparison by the region comparing unit on the characteristic region.13. The characteristic region extracting device according to claim 1,wherein the image data acquiring unit acquires the first image datarepresenting the first image and the second image data representing thesecond image, and the characteristic region defining unit defines thecharacteristic region for the first image and the second imagerespectively, and which further comprises a region comparing unit tocompare the pixel in the characteristic region extracted in the firstimage with the pixel in the characteristic region and its neighboringregions extracted from the second image.
 14. A characteristic regionextracting device which comprises an image data acquiring unit toacquire the first image data representing the first image and the secondimage data representing the second image, said image data representingan image with pixels arranged in a dot matrix pattern, an edge pixeldetecting unit to detect the edge pixels of the image according to thefirst image data and the second image data, a characteristic pointextracting unit to extract the characteristic point in the first imageand the characteristic point in the second image when a pattern formedby the detected edge pixels and their neighboring pixels is similar to aprescribed object to be extracted, an arrangement pattern data creatingunit to create the arrangement pattern data which indicates thearrangement pattern based on the extracted characteristic point in thefirst image, and a characteristic region defining unit to reference thethus created arrangement pattern data and define as the characteristicregion the region in which the characteristic point in the second imageapproximately coincides with the arrangement pattern.
 15. Thecharacteristic region extracting device according to claim 14, whereinthe arrangement pattern data is the data which specifies the relativeposition of the characteristic point.
 16. A characteristic regionextracting method which comprises an image data acquiring step toacquire the image data representing a first image with pixels arrangedin a dot matrix pattern, an edge pixel detecting step to detect the edgepixels of the image according to the image data, a characteristic pointextracting step to extract the edge pixel as the characteristic pointwhen the pattern formed by the detected edge pixels and theirneighboring pixels is similar to a prescribed object to be extracted,and a characteristic region defining unit to define as thecharacteristic region the prescribed region in the image which has theextracted characteristic points in large number.
 17. A characteristicregion extracting program which temporarily stores image datarepresenting an image in a prescribed storage medium and extract acharacteristic region from the image, said program allowing a computerto realize an edge pixel detecting function to detect the edge pixels ofthe image according to the image data stored in the storage medium, acharacteristic point extracting function to extract the edge pixel asthe characteristic point when the pattern formed by the detected edgepixels and their neighboring pixels is similar to a prescribed object tobe extracted, and a characteristic region defining function to define asthe characteristic region the prescribed region in the image which hasthe extracted characteristic points in large number.
 18. Acharacteristic region extracting method which comprises an image dataacquiring step to acquire the first image data representing the firstimage and the second image data representing the second image, saidimage data representing an image with pixels arranged in a dot matrixpattern, an edge pixel detecting step to detect the edge pixels of theimage according to the first image data and the second image data, acharacteristic point extracting step to extract the characteristic pointin the first image and the characteristic point in the second image whena pattern formed by the detected edge pixels and their neighboringpixels is similar to a prescribed object to be extracted, an arrangementpattern data creating step to create the arrangement pattern data whichindicates the arrangement pattern based on the extracted characteristicpoint in the first image, and a characteristic region defining step toreference the thus created arrangement pattern data and define as thecharacteristic region the region in which the characteristic point inthe second image approximately coincides with the arrangement pattern.19. A characteristic region extracting program which temporarily storesin a prescribed storage medium the first image data representing thefirst image and the second image data representing the second image andextracts a characteristic region from the second image which can besuperposed on part of the first image, said program allowing a computerto realize an edge pixel detecting function to detect the edge pixels ofthe image according to the first image data and the second image datastored in the storage medium, a characteristic point extracting functionto extract the characteristic point in the first image and thecharacteristic point in the second image when a pattern formed by thedetected edge pixels and their neighboring pixels is similar to aprescribed object to be extracted, an arrangement pattern data creatingfunction to create the arrangement pattern data which indicates thearrangement pattern based on the extracted characteristic point in thefirst image, and a characteristic region defining function to referencethe thus created arrangement pattern data and define as thecharacteristic region the region in which the characteristic point inthe second image approximately coincides with the arrangement pattern.